\section{Diversity} 
\label{sec:diversity} 


\subsection{Topology Diversity}
\label{sec:topology} 

Network topology illustrates the layout pattern of interconnections of the various elements (links, nodes/ computers, etc) associated with an experiment. The interception of topologies created by various researchers on testbeds can help us understand testbed limitations and topology prevalence.

We adopt the general definitions of \textit{line}, \textit{star}, \textit{ring}, \textit{dumbbell}, \textit{tree} as in \cite{topologies}. For convenience, we define a \textit{tree} to be a complete, perfect tree; i.e., each level (except the last) is completely filled, and all leaves have the same depth/levels, but we do not limit the tree degree. In DETER, we have observed substantial quantity of topologies that have a strongly connected component -- a core -- and then zero or more \textit{satellites} which are lines starting from the core and ending in a LAN or a single node. We classify these further as \textit{core$_0$} with no satellites, 
\textit{core$_1$} with one satellite and \textit{core$_N$} with two or more satellites. We also have a\textit{LAN}, \textit{disconnected LANs} and \textit{disconnected nodes} categories.  A \textit{LAN} stands for a category in which several nodes connect to a centralized switch/router, while \textit{disconnected LANs} indicates that more than one LAN exist in a topology but they are not connected. In \textit{disconnected nodes} topology, there is no LAN but there is at least one node present. 


An intuitive mapping from topology classification to problem solvers would be graph classifiers in the realms of machine learning and pattern recognition, such as k-NN \cite{kNN}, LEAP \cite{LEAP}, etc. However, these methodologies cannot be applied to our problem directly, since they either require the same number of nodes in graphs or labeled graphs. Instead, we utilize graph isomorphism \cite{graphiso} concept to compare each graph candidate against various topology definitions. Therefore our classification problems becomes determining whether two finite graphs are isomorphic. Please note that, graphs in our work are understood to be undirected, non-labeled and non-weighted graphs. We use  igraph \cite{igraph} algorithm to detect isomorphism. .
\begin{figure*}[t] 
\begin{center} 
\subfigure[Experiments]{
	\includegraphics[width=2in, type=pdf,ext=.pdf,read=.pdf]
	{figs/topo.gnu} 
	\label{fig:topoe}
	}
	\subfigure[Projects]{
	\includegraphics[width=2in, type=pdf,ext=.pdf,read=.pdf]
	{figs/topop.gnu} 
	\label{fig:topop}
	}
\caption{Experiments and projects with a given
topology type in DETER} \label{topo}
\end{center} \end{figure*}

The total number of topologies we analyzed in DETER is 2,775, and about 10\% of those we could not classify. As discussed in Section \ref{sec:term} these are a subset of all topologies that existed on the testbed. We lose records of topologies if  the corresponding experiment definition changed or was deleted. Figure \ref{fig:topoe} shows the distributions of those topologies we could classify in DETER across research no-outcome, research outcome and class outcome categories. The x-axis orders topologies by complexity. All categories show similar trends. We note high popularity of simple topologies such as disconnected nodes, a LAN and a line. Regular topologies such as ring, star, tree and dumbbell are rarely used. Complex topologies resembling realistic network topologies are also popular, such as core$_1$ and core$_N$. We note that distributions here are likely skewed because some projects may generate many more experiment definitions (and thus topologies) than others. Figure \ref{fig:topop} corrects for this bias by showing distribution of projects that use a specific topology (one or more times). We again note high popularity of simple and complex topologies. Simple topologies are used by 28-67\% of projects. For example, 67\% of research outcome projects use a line topology. Complex topologies are used by 20-64\% of projects. For example, 56\% of research outcome projects uses a core$_N$ topology.



\subsection{Project Diversity}
\label{sec:project}
\begin{figure*} 
\begin{center} 
	\subfigure[Experiment Size]{
	\includegraphics[width=2in, type=pdf,ext=.pdf,read=.pdf]
	{figs/projdiv1} 
	\label{fig:projdiv1}
	}
	\subfigure[Experiment Duration]{
	\includegraphics[width=2in, type=pdf,ext=.pdf,read=.pdf]
	{figs/projdiv2} 
	\label{fig:projdiv2}
	}
	\subfigure[Topology Complexity]{
	\includegraphics[width=2in, type=pdf,ext=.pdf,read=.pdf]
	{figs/projdiv3}
	\label{fig:projdiv3}
	}
\caption{Research outcome project diversity on DETER}
\label{projdiv}
\end{center}
\end{figure*}

We examine diversity of projects with regard to the sizes, duration and topologies of their experiment instances, with the goal to understand project dynamics and evolution. We only perform this analysis for research outcome projects on DETER, because there we can establish the fact that the testbed was useful for attaining a research goal. We classify experiment instance sizes as small ($<=3$ nodes), medium size ($4-20$ nodes) and large ($>20$ nodes). We classify durations as short ($<=1$ hour), medium ($>1$ hour but $<=1$ day) and long ($>1$ day). We classify topologies as simple (disconnected nodes, LAN, line and disconnected LANs), regular (ring, star, tree and dumbbell) and complex (core$_0$, core$_1$ and core$_2$). 

Figure~\ref{projdiv} shows the distribution of projects across these three dimensions (size, duration and topology complexity) and our categories using Venn diagrams. There are 32 projects that create experiments in all three size categories, 10 create only small and medium-size experiments and 1 creates only medium-size and large experiments. There are 2 projects that create only medium-size topologies, and no project has only small or only large topologies. Presence of many projects in more than one size category indicates that different-sized topologies serve different purpose in experimentation and are needed by many projects. Small topologies are suitable for quick trials of new ideas. Medium-sized topologies facilitate deeper exploration and measurement, and large-sized topologies serve to prove scalability. 

Looking into duration of experiment instances there are 38 projects with short, medium and long duration instances. Two projects have only short and medium duration instances, and three have only medium and long duration instances. There are 2 projects that only have medium duration instances. No project has only short or only long instances. This presence of many projects in more than one duration-category indicates different purpose of different-duration instances. Short instances may work on a simple task or may indicate trial and error on the user side. Medium duration instances indicate user-interactive experimentation, while long duration instances indicate either scripted, parameter space exploration or an experiment with a complex and lengthy setup which should not be automatically swapped out by the testbed.

Looking into complexity there were three projects that only used topologies we could not classify and we take them out of consideration. We note a different trend here than at size and duration dimension. All but one project use simple topologies, as can be expected because such topologies facilitate quick trial of new ideas. But there are 11 projects (more than a quarter of our working set for this analysis) that only use simple topologies, which means that such topologies facilitate meaningful security research in spite of their simplicity.  Regular topologies are used by 16 projects and complex topologies by 27 projects. High presence of projects in intersection with the complex category indicate a likely progression of experimentation from simple topologies where users develop and validate their ideas, to more complex but harder to create and debug topologies where users perform extensive measurements of a mature prototype in a realistic setting.


We illustrate the dynamics of a select research outcome project in Figure~\ref{fig:dyn}. Both subfigures show experiment instances over time with circles, where the diameter of the circle corresponds to the instance duration. Since many instances are too short to be visible (project spans five years) we also show the start of an instance at a given time with a grey square.
Vertical lines note publication times. 
Figure \ref{fig:tva2} shows the instance size on the y axis. We note the progression of experimentation from smaller to larger instances (seen as circles on an almost vertical line) and from shorter to longer ones (seen as circles on a horizontal line with expanding diameter), with occasional return to small and short instances, possibly to test new ideas. 
Figure \ref{fig:tva} shows the topology type on the y axis, ordered by complexity. We again note the progression of experimentation from simpler to more complex instances (seen as circles on an almost vertical line). We see intensive experimentation with star and core$_N$ topologies -- many instances and longer durations, possibly to generate data for a publication. We also see occasional return to simple topologies such as line, LAN or disconnected nodes for a quick measurement or to try new ideas. Topologies core$_0$, tree and dumbbell were tried and quickly abandoned.

\begin{figure*}
\begin{center} 
	\subfigure[Size]{
	\includegraphics[width=2.3in,  angle=90, type=pdf,ext=.pdf,read=.pdf]
	{figs/tva2.gnu} 
		\label{fig:tva2}
	}
	\subfigure[Topology]{
	\includegraphics[width=3in, type=pdf,ext=.pdf,read=.pdf]
	{figs/tva.gnu} 
		\label{fig:tva}
	}
\caption{Evolution of size, duration and complexity of experiment instances over a life time of a research outcome project}
\label{fig:dyn}
\end{center}
\end{figure*}


